Tissue Characterization Based on Machine Learning in Medical Imaging

ABSTRACT

Tissue is characterized using machine-learnt classification. The prognosis, diagnosis or evidence in the form of a similar case is found by machine-learnt classification from features extracted from frames of medical scan data. The texture features for tissue characterization may be learned using deep learning. Using the features, therapy response is predicted from magnetic resonance functional measures before and after treatment in one example. Using the machine-learnt classification, the number of measures after treatment may be reduced as compared to RECIST for predicting the outcome of the treatment, allowing earlier termination or alteration of the therapy.

BACKGROUND

The present embodiments relate to tissue characterization in medicalimaging.

Magnetic resonance images are widely used in medical diagnosis andtherapy. For example, magnetic resonance is used for breast tumordiagnosis following the guidelines of the Breast Imaging-Reporting andData System (BIRADS), which are based on clinically descriptive tagslike mass (shape, margin, mass enhancement), symmetry or asymmetry,non-mess-like enhancement in an area that is not a mass (distributionmodifiers, internal enhancement), kinetic curve assessment, and otherfindings. Similarly for prostate, the Prostate Imaging and Reporting andData System (PIRADS) specifies the clinically descriptive tags forspecial prostate regions, such as peripheral zone, central zone, andtransition zone. For liver tissue characterization, fibrosis staging ispossible based on reading of the magnetic resonance images. Similarapproaches are used in other imaging modalities, such as ultrasound,computed tomography, positron emission tomography, or single photonemission computed tomography.

To assess therapy, multimodal magnetic resonance scans are acquiredbefore and after therapy. A simple morphological (e.g., size-based)scoring is commonly performed in tumor treatment assessment, such as theResponse Evaluation Criteria in Solid Tumors (RECIST) criteria. Theassessment of treatment response is critical in determining the courseof continuing treatment since chemotherapy drugs may have adverseeffects on the patient. In basic clinical settings, treatment assessmentis done morphologically with tumor size. Due to this simple approach, itcan take longer to determine if a treatment is succeeding.

The decision to stop therapy may occur earlier by employing functionalmagnetic resonance information than with the RECIST criteria. Forexample, treatment effectiveness may be determined earlier by usingimage-based functional measurements, such as intensity based histogramsof the functional measures. These histogram-based intensity values aremanually analyzed in clinical practice and may not necessarily capturesubtleties related to image texture and local dissimilarity that maybetter represent cell density, vasculature, necrosis, or hemorrhagecharacteristics important to clinical diagnosis.

BRIEF SUMMARY

By way of introduction, the preferred embodiments described belowinclude methods, systems, instructions, and non-transitory computerreadable media for tissue characterization in medical imaging. Tissue ischaracterized using machine-learnt classification. The prognosis,diagnosis or evidence in the form of a similar case is found bymachine-learnt classification from features extracted from frames ofmedical scan data with or without external data such as age, gender, andblood biomarkers. The texture or other features for tissuecharacterization may be learned using deep learning. Using the features,therapy response is predicted from magnetic resonance functionalmeasures before and after treatment in one example. Using themachine-learnt classification, the number and time between measuresafter treatment may be reduced as compared to RECIST for predicting theoutcome of the treatment, allowing earlier termination or alteration ofthe therapy.

In a first aspect, a method is provided for tissue characterization inmedical imaging. A medical scanner scans a patient where the scanningprovides multiple frames of data representing a tissue region ofinterest in the patient. A processor extracts values for features fromthe frames of data. A machine-learnt classifier implemented by theprocessor classifies a therapy response of the tissue of the tissueregion from the values of the features as input to the machine-learntclassifier. The therapy response is transmitted.

In a second aspect, a method is provided for tissue characterization inmedical imaging. A medical scanner scans a patient where the scanningprovides different frames of data representing different types ofmeasurements for tissue in the patient. A processor extracts values forfeatures from the frames of data. A machine-learnt classifierimplemented by the processor classifies the tissue of the patient fromthe values of the features as input to the machine-learnt classifier.The tissue classification is transmitted.

In a third aspect, a method is provided for tissue characterization inmedical imaging. A patient is scanned with a medical scanner where thescanning provides a frame of data representing a tumor in the patient. Aprocessor extracts values for deep-learnt features from the frame ofdata. A deep-machine-learnt classifier implemented by the processorclassifies the tumor from the values of the features as input to amachine-learnt classifier. The classification of the tumor istransmitted.

The present invention is defined by the following claims, and nothing inthis section should be taken as a limitation on those claims. Furtheraspects and advantages of the invention are discussed below inconjunction with the preferred embodiments and may be later claimedindependently or in combination.

BRIEF DESCRIPTION OF THE DRAWINGS

The components and the figures are not necessarily to scale, emphasisinstead being placed upon illustrating the principles of the invention.Moreover, in the figures, like reference numerals designatecorresponding parts throughout the different views.

FIG. 1 is a flow chart diagram of one embodiment of a method forcharacterizing tissue in medical imaging;

FIG. 2 represents user interaction for tissue characterization accordingto one embodiment;

FIG. 3 is a flow chart diagram of a machine-learnt classifier embodimentof the method for characterizing tissue in medical imaging;

FIG. 4 is a flow chart diagram of a deep-learning embodiment of themethod for characterizing tissue in medical imaging;

FIG. 5 is an example of a deep-learning convolution layer-basedclassification network incorporating non-imaging features;

FIG. 6 illustrates an example output of a related case identifiedthrough classification;

FIG. 7 illustrates four example applications of machine-learntclassification of tissue characteristics;

FIG. 8A shows an example time line for measuring therapy responserelying on trending over many times post treatment, and FIG. 8B shows anexample time line for measuring the therapy response relying onmeasurement for just one time post treatment; and

FIG. 9 is one embodiment of a system for tissue characterization inmedical imaging.

DETAILED DESCRIPTION OF THE DRAWINGS AND PRESENTLY PREFERRED EMBODIMENTS

In advanced image analytics, complex features, such as image texture,are automatically found and included in an automated format capable ofbeing employed in clinical use cases. Using machine-learntclassification to interrogate the image has the potential to extracttexture and/or other image-based information for classification inclinical use cases. A generic pipeline and user interface is capable offinding and processing complex features from medical image data.

Several approaches for tumor prognosis and outcomes allow such analyticsalong with an interface to make the analytics readily accessible tophysicians and support staff. The first approach uses robust imagetextural and/or other image features extracted by the computer as afeature set for further analysis. The second approach uses a deeplearning pipeline involving a Siamese network to automatically createand classify features in a parallel convolutional network for the samepatient at two different time points and/or different types of measures.The deep learning network and interface are provided for image analyticsfor lesion and/or tissue characterization and therapy prediction.

In one embodiment for tumor prognosis, diagnosis or finding of similarcases, a series of features is extracted from multiple image contrastsand/or multiple examination time-points. The textural and/or other imagefeatures are used in a classifier. A Siamese deep-learning neuralnetwork may be used to identify the features. Non-image data, such bloodtest results age, gender, or blood serum biomarkers, may also be used asfeatures. The extracted features are computed against each other using amachine-learnt classifier to determine the diagnosis, prognosis, or findsimilar cases. For finding similar cases, the images from the similarcases are obtained from a database of previously document cases. Thepreviously documented cases are used to determine the reference case orcases. The diagnosis, prognosis, or finding of similar cases may beperformed via a cloud based service that delivers results.

For the user interface, a user marks the tumor of interest in one ormore images of a time series, possibly with a single click, and thecomputer produces the diagnosis, prognosis, or similar cases report inresponse. This clinical product and interface may minimize interaction.The single click per image assists in segmentation and/or more preciseidentification of regions. By only requiring a single click or regiondesignation in the pre and post treatment images, the interaction isminimized, increasing the clinical feasibility of the approach. Theoutput report may include clinical evidence, in the form of numbers(e.g., quantifying the textural features that lead to the decision) orin the form of references to previous clinical cases. A single-clickinterface determines results, allowing for a clinically feasibleapproach to incorporate complex image features into tumor prognosis andtherapy.

FIG. 1 shows one embodiment of a flow chart of a method for tissuecharacterization in medical imaging. For tissue characterization, suchas tumor type, tumor response to treatment, identification of similartumors in other patients, tumor prognosis, or tumor diagnosis, amachine-learnt classifier is applied. The machine-learnt classifier usesinformation from one or more frames of data to classify the tumor.Frames of data from different times or different types of measures maybe used to classify. In one approach, the features extracted from theframes as the information are manually defined. In another approach,deep learning identifies the features that best characterize the tumor.The features learned with deep learning may be texture and/ornon-texture, such as features from the frame or clinical data.

The acts are performed in the order shown (e.g., top to bottom) or otherorders. Additional, different, or fewer acts may be provided. Forexample, the method is performed without transmitting the classificationin act 20. As another example, the segmentation of act 14 is notperformed, instead the classification is applied to the entire frame ofdata.

In act 12, one or more medical images or datasets are acquired. Themedical image is a frame of data representing the patient. The data maybe in any format. While the terms “image” and “imaging” are used, theimage or imaging data may be in a format prior to actual display of theimage. For example, the medical image may be a plurality of scalarvalues representing different locations in a Cartesian or polarcoordinate format different than a display format. As another example,the medical image may be a plurality red, green, blue (e.g., RGB) valuesoutput to a display for generating the image in the display format. Themedical image may not yet be a displayed image, may be a currentlydisplayed image, or may be previously displayed image in the display orother format. The image or imaging is a dataset that may be used forimaging, such as scan data representing the patient.

Any type of medical image may be used. In one embodiment, magneticresonance frames of data representing a patient are acquired. Magneticresonance data is acquired by scanning with a magnetic resonance system.Using an imaging sequence, the magnetic resonance system scans thepatient. Data representing an interior region of a patient is acquired.The magnetic resonance data is k-space data. Fourier analysis isperformed to reconstruct the data from the k-space into athree-dimensional object or image space, providing the frame of data. Inother embodiments, x-ray, computed tomography, ultrasound, positronemission tomography, single photon emission computed tomography, orother medical imaging scanner scans the patient. Combination scanners,such as magnetic resonance and positron emission tomography or computedtomography and positron emission tomography systems may be used to scan.The scan results in a frame of data acquired by the medical imagingscanner and provided directly for further processing or stored forsubsequent access and processing.

The frame of data represents a one, two, or three-dimensional region ofthe patient. For example, the multi-dimensional frame of data representsan area (e.g., slice) or volume of the patient. Values are provided foreach of multiple locations distributed in two or three dimensions. Atumor or suspicious tissue within the patient is represented by thevalues of the frame of data.

The frame of data represents the scan region at a given time or period.The dataset may represent the area or volume over time, such asproviding a 4D representation of the patient. Where more than one frameis acquired, the different frames of data may represent the same oroverlapping region of the patient at different times. For example, oneor more frames of data represent the patient prior to treatment, and oneor more frames of data represent the patient after treatment orinterleaved with on-going treatment.

Where more than one frame is acquired, the different frames mayrepresent different contrasts. For example, different types of contrastagents are injected or provided in the patient. By scanning tuned to orspecific to the different types of contrast, different frames of datarepresenting the different contrast agents are provided.

Where more than one frame is acquired, the different frames mayrepresent different types of measures (multi-modal or multi-parametricframes of data). By configuring the medical scanner, different types ofmeasurements of the tissue may be performed. For example in magneticresonance, both anatomical and functional measurements are performed. Asanother example in magnetic resonance, different anatomical or differentfunctional measurements are performed. For different anatomicalmeasurements, T1 and T2 are two examples. For different functionalmeasurements, apparent diffusion coefficient (ADC), venous perfusions,and high B-value are three examples. In one embodiment, a T2 frame ofdata and an ADC frame of data are computed (e.g., different b-valueimages are acquired to compute a frame of ADC data). Frames of data fromdifferent types of scanners may be used.

A combination of different times and types of measures may be used. Forexample, one set of frames of data represents different types ofmeasures (e.g., T2 and ADC) for pre-treatment, and another set of framesof data represent the same different types of measures (e.g., T2 andADC) for post-treatment. Multi-dimensional and multi-modal image data isprovided for each time. In other embodiments, a single frame of datarepresenting just one type of measure for one time is acquired byscanning.

Where multiple frames represent the tissue at different times, theframes are spatially registered. The registration removes translation,rotation, and/or scaling between the frames. Alternatively, registrationis not used.

In act 14, the tissue of interest is identified. The tissue of interestis identified as a region around and/or including the tumor. Forexample, a box or other shape that includes the tumor is located.Alternatively, the tissue of interest is identified as tissuerepresenting the tumor, and the identified tumor tissue is segmented forfurther analysis.

The identification is performed by the user. The user, using a userinput (e.g., mouse, trackball, keyboard, buttons, sliders, and/or touchscreen), identifies the tissue region to be used for feature extraction,classification, and transmission of the classification results. Forexample, the user selects a center of the tumor about which a processorplaces a box or other region designator. The user may size or position aregion designator with or without center selection. In other approaches,the user indicates a location on the suspicious tissue, and theprocessor segments the suspicious tissue based on the user placed seed.Alternatively, a processor automatically identifies the tissue region ofinterest without user selection.

FIG. 2 shows an example graphic user interface or approach with minimaluser interaction for tissue characterization. In the example of FIG. 2,the classification is for reporting on therapy. Two images correspondingto two frames of data at two times are provided, one pre-treatment andone post-treatment. Given both the pre and post treatment image sets,the user clicks on the tumor center (represented by the arrow tip) ineach time point. A bounding box centered at the user selected point isplaced, designating the tissue region of interest.

In response to the selection, the computer then returns the predictedsuccess of treatment in an automated report. The report may also containdiagnosis of similar cases that have been previously reviewed in adatabase as references. The treatment outcome or other classification isdetermined via a single-click on each image or frame of data without theneed to perform any manual and/or automatic segmentation. A“single-click” or simple user input is provided for tumor diagnosis,treatment planning, and/or treatment response assessment. The sameapproach and technology can be used in any medical imaging product thatexamines tumor prognosis and treatment based on data acquired from ascanner.

The tumor tissue with or without surrounding tissue is segmented. Thedata is extracted from the frame for further processing. The pixel orvoxel values for the region of interest are isolated. Alternatively, thelocations in the region are flagged or marked without being separatedfrom other locations in the frame of data.

In act 16 of FIG. 1, a processor extracts values for features. Theprocessor is part of the medical imaging scanner, a separateworkstation, or a server. In one embodiment, the processor extractingthe values is a server remote from the medical scanner, such as a serverin the cloud. A manufacturer of the medical scanner or a third partyprovides classification as a service, so the frames of data arecommunicated through a computer network to the server for extraction ofthe features and classification from the extracted values.

Values for any number of features are extracted from the frame or framesof data. The values for a texture of the tissue represented by at leastone of the frames of data are extracted. The texture of the tissue isrepresented by the measures of the frame of data. The extraction of thevalues for each feature is performed for the tissue region of interest,avoiding application to other tissue outside the region of interest.Alternatively, the values for other regions outside the region ofinterest are extracted.

Each feature defines a kernel for convolution with the data. The resultsof the convolution are a value of the feature. By placing the kernel atdifferent locations, values for that feature at different locations areprovided. Given one feature, the values of that feature at differentlocations are calculated. Features for other texture information thanconvolution may be used, such as identifying a maximum or minimum. Otherfeatures than texture information may be used.

In one embodiment, the features are manually designed. The feature orfeatures to be used are pre-determined based on a programmer'sexperience or testing. Example features include scaled invariant featuretransformation, histogram of oriented gradients, local binary pattern,gray-level co-occurrence matrix, Haar wavelets, steerable, orcombinations thereof. A feature extraction module computes features fromimages to better capture essential subtleties related to cell density,vasculature, necrosis, and/or hemorrhage that are important to clinicaldiagnosis or prognosis of tissue.

FIG. 3 shows an example using manually programmed features. One or moreimages are acquired from memory or directly from a scanner. The texturalfeatures are extracted from the images. The values of the features areused for classification to provide outcomes.

In another embodiment, deep-learnt features are used. The values areextracted from frames of data for features learned from machinelearning. Deep machine learning learns features represented in trainingdata as well as training the classifier rather than just training theclassifier from the manually designated features.

Any deep learning approach or architecture may be used. In oneembodiment, the extracting and classifying of acts 16 and 18 are basedon a twin or Siamese convolution network. FIG. 4 shows an example fordeep learning. The twin or Siamese convolutional networks are trained toextract features from given multi-modal, multi-dimensional images, IM1and IM2, such as multi-modal frames representing pre and post treatment,respectively. The relevant features are automatically determined as partof training. This ability allows for the generic training on arbitrarydata (i.e., training data with known outcomes) that can internallydetermine features, such as textures. The Siamese convolution networksare linked so that the same weights (W) for the parameters defining thenetworks are used in both branches. The Siamese network uses two inputimages (e.g., one branch for pre-treatment and another branch forpost-treatment). Kernels weights for convolution are learnt using bothbranches of the network, and optimized to provide values to Gw.

The Siamese deep learning network is also trained to classify small,large or absence of changes between time points. The definition of suchfeatures is based on a specific loss function Ew that minimizesdifference between time points when there is no or small changes andmaximizes the differences when there are large changes between them. Thefeatures indicating relevant differences between the two inputs arelearned from the training data. By training the network with labeledoutcomes, the network learns what features are relevant or can beignored for determining the prognosis, diagnosis, or finding similarcases. During training, low-level features and invariants are learned bythe convolutional networks that have exactly the same parameters. Thesenetworks determine the core feature set that differs between the twoinput datasets based on feedback during learning of the differencenetwork.

FIG. 5 shows an example convolution layer-based network for learning toextract features. Each branch or twin has the same layers or networkstructure. The networks themselves are shown as layers of convolutional,sub-sampling (e.g., max pooling), and fully connected layers. By usingconvolution, the number of possible features to be tested is limited.The fully connected layers (FC) in FIG. 5 operate to fully connect thefeatures as limited by the convolution layer (CL) after maximum pooling.Other features may be added to the FC layers, such as non-imaging orclinical information. Any combination of layers may be provided.Additional, different, or fewer layers may be provided. In onealternative, a fully connected network is used instead of a convolutionnetwork.

Returning to FIG. 4, the two parallel networks process the pre and posttherapy data, respectively. The networks are trained with exactly thesame parameters in the Siamese network. The features that optimallydiscriminate based on the loss function, Ew, are automatically developedduring training of the network. For example, the multi-modal inputframes are for T2 and ADC for each time. The features related totextural information for the T2 image and local deviations in the ADCimage highlighting the differences from pre and post treatment arelearned. These learnt features are then applied to frames of data for aspecific patient, resulting in Gw values for the specific patient.

In act 18 of FIG. 1, the machine-learnt classifier classifies the tissueof the patient from the extracted values of the features. The values areinput to the machine-learnt classifier implemented by the processor. Byapplying the classifier, the tissue is classified. For example, atherapy response of the tissue in the tissue region is classified fromthe values of the features as input to the machine-learnt classifier.

In the approach of FIG. 3, any machine-learnt classifier may be used.The classifier is trained to associate the categorical labels (output)to the extracted values of one or more features. The machine-learning ofthe classifier uses training data with ground truth, such as values forfeatures extracted from frames of data for patients with known outcomes,to learn to classify based on the input feature vector. The resultingmachine-learnt classifier is a matrix for inputs, weighting, andcombination to output a classification. Using the matrix or matrices,the processor inputs the extracted values for features and outputs theclassification.

Any machine learning or training may be used. A probabilistic boostingtree, support vector machine, neural network, sparse auto-encodingclassifier, Bayesian network, or other now known or later developedmachine learning may be used. Any semi-supervised, supervised, orunsupervised learning may be used. Hierarchal or other approaches may beused.

In one embodiment, the classification is by a machine-learnt classifierlearnt with the deep learning. As part of identifying features thatdistinguish between different outcomes, the classifier is also machinelearnt. For example in FIG. 4, the classifier 18 is trained to classifythe tissues based on the feature values Gw, obtained from a Siamesenetwork that is already optimized/trained to maximize differences ofimages from different categories and minimize differences of images fromthe same categories. For example, the classifier categorizes the tumorfrom the feature values, such as classifying a type of tumor, a tumorresponse to therapy, or other tissue characteristic. In the example ofFIG. 4, the classification is based on features from Gw, which areoptimized from the loss Ew. First, the Siamese network trains and/ordefines kernels such that the feature vectors Gw can help discriminatefor all categories. Once, the network is trained, the classifier 18 usesGw and then defines probabilities that two images from different timepoints have zero, small or large differences.

In either approach (e.g., FIG. 3 or FIG. 4), additional information maybe used for extracting and/or classifying. For example, values ofclinical measurements for the patient are used. The classifier istrained to classify based on the extracted values for the features inthe frames of data as well as the additional measurements. Genetic data,blood-based diagnostics, family history, sex, weight, and/or otherinformation are input as a feature for classification.

The classifier is trained to classify the tumor. The classifier istrained to classify the tissue into one of two or more classes. Byinputting extracted values for a specific patient, the machine-learntclassifier classifies the tumor for that patient into one of theclasses. Any of various applications may be used.

In one embodiment, the classifier identifies a similar case. The similarcase includes an example treatment and outcome for another patient witha tissue region similar to the tissue region of the current patient. Anynumber of these reference cases may be identified. A database ofpossible reference cases is used. The most similar case or cases to thecurrent patient are identified by the classifier. Using the extractedvalues for texture features with or without other features, theclassifier identifies the class as a reference case or cases. Inevidence-based medicine, decision-making is optimized by emphasizing theuse of evidence from well designed and conducted research. One keycomponent is to retrieve the evidence in the form of other cases usingthe current case. For example, as in FIG. 6, once a lesion is marked bythe user, the values for the textural features are extracted and used bythe classifier to retrieve the closest cases. These closest casesprovide the evidence, such as a list of similar cases with thumbnailimages of the tumors for the reference cases.

Another class is therapy response. The success or failure of therapy ispredicted as a diagnosis or prognosis. In an alternative, rather thanbinary indication of success or failure, a range providing an amountand/or probability of success or failure is output as the class. Whetherthe patient is likely to respond (i.e., responder), not likely torespond (i.e., non-responder), or may partially respond (i.e., partialor semi-responder) is output. The predicted survival time of the patientmay be the output.

FIG. 7 shows two examples of classifying the tissue as successful or notsuccessful therapy. In applications #3 and #4, the therapy response ofthe tumor is classified. A trans arterial chemo embolization (TACE) isused as the example therapy. In application #3, multi-parametric (e.g.,T2 and ADC) data for one time (e.g., 1 month after treatment) is used.After identifying the tissue region of interest for the tumor, theclassification is applied to determine whether the tumor is respondingto the treatment. The level or rate of response may be output, informinga decision on any continued treatment and level of treatment. In analternative, treatment is not applied. The frames of data prior totreatment are used to classify whether the treatment is expected to besuccessful based on the extracted textural and/or other features priorto treatment.

In application #4, multiple treatments are performed. Frames of dataafter each treatment are used. The response of the tumor is measured orpredicted based on the classification from the data over time. In analternative, only the second treatment is performed and the first framesof data at 1 month are pre-treatment frames. The features from thedifferent times are used to predict or measure therapy response.

By inferring the therapy success or level of success for therapy appliedto the tissue region, a decision on whether to continue therapy and/orto change the therapy may be more informed and/or performed earlier.FIG. 8A shows an example of predicting therapy outcome. In this example,RECIST is used, so the size change of the tumor is measured N timespost-therapy. Eventually, a sufficient trend is determined by theclinician to predict the outcome. FIG. 8B shows an alternative approachusing the machine-learnt classifier. Less information is needed, such asjust one set of post therapy frames of data (e.g., from one time orappointment for scanning). With textural and/or other feature analysisand machine-learnt classification, the therapy success or failuredecision may be inferred without manual perception of trends. A lessernumber of scans and lesser amount of time are needed to make therapydecisions. The success of the therapy applied to the tumor is inferredand used to optimize treatment.

Other classes for machine-learnt classification may be used. Theclassifier may be machine trained to classify the tumor (e.g.,suspicious tissue region) as benign or malignant. Once the lesion issegmented or identified, values for textural features are computed forthe lesion. The values are fed into the machine-learnt classifier forlabeling of malignancy.

The classifier may be trained to output values for staging the tumor.Using advanced tissue characterization provided by the machine-learntclassifier, the stage is output. For example in liver tissue, theextracted textural features are used by the classifier to output ameasure of fibrosis staging. In other examples, the classifier istrained to output tags used for staging, such as outputting the measuresused for staging the tumor. The values for the features are used by theclassifier to provide the tag or staging measure. In quantitative BIRADSfor beast examination, the textural features are extracted, and then theclassifier associates the categorical labels of clinically descriptivetags (e.g., measures of mass, symmetry, and non-mess-like enhancement)to the extracted features. The inferred tags are then used to manuallyor automatically stage the breast tumor. In quantitative PIRADS forprostate examination, the textural features are extracted, and then theclassifier associates the categorical labels of clinically descriptivetags (e.g., tags for the peripheral zone, central zone, and transitionzone) to the extracted features. The inferred tags are then used tomanually or automatically stage the prostate.

In another embodiment, the classifier is trained to output anyinformation useful for diagnosis or prognosis. For example, informationto enhance therapy monitoring is output. An intensity histogram,histogram of difference over time in the intensities representing thetumor, and/or a difference of histograms of intensities representing thetumor at different times are calculated and output without theclassifier. The classifier supplements these or other image intensitystatistics or histograms. Information derived from the textual featuresand/or other features is used to provide any information useful toclinicians.

More than one machine-trained classifier may be used. The same ordifferent features are used by different classifiers to output the sameor different information. For example, a classifier is trained forpredicting therapy response, and another classifier is trained to outputtags for staging. In alternative embodiments, one classifier is trainedto output different types of information, such as using a hierarchalclassifier.

FIG. 7 shows four example use cases of classification usingmulti-parametric magnetic resonance frames of data. “Localize Lesion”represents identifying a tissue region of interest (ROI) by segmentationor by user region designation (e.g., act 14). In applications #1, #2,and #3, the input to the system is a set of multi-parametric magneticresonance images from a single time point, and the system performs oneor more of three different tasks: in application #1, the task is topredict whether the tumor is benign or malignant. The outcome may beeither discrete (e.g., Yes vs No) or continuous (e.g., giving anindication of the severity of the malignancy, such as a number between 0and 1). In application #2, the task is to predict whether the tumor islow- or high-grade. The grading may be discrete (e.g., Low vs High) orcontinuous (e.g., a number between 0 to the N, where N is the highestgrade possible). In application #3, the task is to predict whether thepatient responds to the therapy. The response may be discrete (e.g.,responding vs not-responding) or continuous (a number indicating thedegree of response, such as between 0 and 1). In application #4, theinput to the system is a set of multi-parametric magnetic resonanceimages from two or more time points, and the system performs the sametask as in applications #3, which is to predict the response, based onthe values extracted from each of the images.

In act 20 of FIG. 1, the tissue classification is transmitted. Any ofthe tissue classifications output by the classifier are transmitted.Alternatively, information derived from the output of the classificationis transmitted, such as a stage derived from classification of tags.

The transmission is to a display, such as a monitor, workstation,printer, handheld, or computer. Alternatively or additionally, thetransmission is to a memory, such as a database of patient records, orto a network, such as a computer network.

The tissue classification is output to assist with prognosis, diagnosis,or evidence-based medicine. For example, a list of similar patients,including their treatment regime and outcome, is output. As anotherexample, a predicted therapy response is output in a report for thepatient.

The tissue classification is output as text. An image of the tumor isannotated or labeled with alphanumeric text to indicate theclassification. In other embodiments, an image of the tissue isdisplayed, and the classification is communicated as a symbol, coloring,highlighting or other information added onto the image. Alternatively,the classification is output in a report without the image of the tumoror separated (e.g., spaced away) from the image of the tumor.

The tissue may also be classified very locally (e.g., independentclassification of every voxel). The resulting classification is outputas a colored or highlighted overlay onto images of the tissue, visuallyindicating, spatially, possible regions of likely response ornon-response.

Other information may be output as well. Other information includesvalues for the features, clinical measures, values from imageprocessing, treatment regime, or other information (e.g., lab results).

FIG. 9 shows a system for tissue characterization in medical imaging.The system includes an imaging system 80, a memory 84, a user input 85,a processor 82, a display 86, a server 88, and a database 90.Additional, different, or fewer components may be provided. For example,a network or network connection is provided, such as for networking witha medical imaging network or data archival system. In another example,the user input 85 is not provided. As another example, the server 88 anddatabase 90 are not provided. In other examples, the server 88 connectsthrough a network with many imaging systems 80 and/or processors 82.

The processor 82, memory 84, user input 85, and display 86 are part ofthe medical imaging system 80. Alternatively, the processor 82, memory84, user input 85, and display 86 are part of an archival and/or imageprocessing system, such as associated with a medical records databaseworkstation or server, separate from the imaging system 80. In otherembodiments, the processor 82, memory 84, user input 85, and display 86are a personal computer, such as desktop or laptop, a workstation, aserver, a network, or combinations thereof. The processor 82, display86, user input 85, and memory 84 may be provided without othercomponents for acquiring data by scanning a patient.

The imaging system 80 is a medical diagnostic imaging scanner.Ultrasound, computed tomography (CT), x-ray, fluoroscopy, positronemission tomography, single photon emission computed tomography, and/ormagnetic resonance (MR) systems may be used. The imaging system 80 mayinclude a transmitter and includes a detector for scanning or receivingdata representative of the interior of the patient.

In one embodiment, the imaging system 80 is a magnetic resonance system.The magnetic resonance system includes a main field magnet, such as acryomagnet, and gradient coils. A whole body coil is provided fortransmitting and/or receiving. Local coils may be used, such as forreceiving electromagnetic energy emitted by atoms in response to pulses.Other processing components may be provided, such as for planning andgenerating transmit pulses for the coils based on the sequence and forreceiving and processing the received k-space data. The received k-spacedata is converted into object or image space data with Fourierprocessing. Anatomical and/or functional scanning sequences may be usedto scan the patient, resulting in frames of anatomical and/or functionaldata representing the tissue.

The memory 84 may be a graphics processing memory, a video random accessmemory, a random access memory, system memory, cache memory, hard drive,optical media, magnetic media, flash drive, buffer, database,combinations thereof, or other now known or later developed memorydevice for storing data or video information. The memory 84 is part ofthe imaging system 80, part of a computer associated with the processor82, part of a database, part of another system, a picture archivalmemory, or a standalone device.

The memory 84 stores medical imaging data representing the patient,segmentation or tissue region information, feature kernels, extractedvalues for features, classification results, a machine-learnt matrix,and/or images. The memory 84 may alternatively or additionally storedata during processing, such as storing seed locations, detectedboundaries, graphic overlays, quantities, or other information discussedherein.

The memory 84 or other memory is alternatively or additionally anon-transitory computer readable storage medium storing datarepresenting instructions executable by the programmed processor 82 fortissue classification in medical imaging. The instructions forimplementing the processes, methods and/or techniques discussed hereinare provided on non-transitory computer-readable storage media ormemories, such as a cache, buffer, RAM, removable media, hard drive orother computer readable storage media. Non-transitory computer readablestorage media include various types of volatile and nonvolatile storagemedia. The functions, acts or tasks illustrated in the figures ordescribed herein are executed in response to one or more sets ofinstructions stored in or on computer readable storage media. Thefunctions, acts or tasks are independent of the particular type ofinstructions set, storage media, processor or processing strategy andmay be performed by software, hardware, integrated circuits, firmware,micro code and the like, operating alone, or in combination. Likewise,processing strategies may include multiprocessing, multitasking,parallel processing, and the like.

In one embodiment, the instructions are stored on a removable mediadevice for reading by local or remote systems. In other embodiments, theinstructions are stored in a remote location for transfer through acomputer network or over telephone lines. In yet other embodiments, theinstructions are stored within a given computer, CPU, GPU, or system.

The user input 85 is a keyboard, mouse, trackball, touch pad, buttons,sliders, combinations thereof, or other input device. The user input 85may be a touch screen of the display 86. User interaction is received bythe user input, such as a designation of a region of tissue (e.g., aclick or click and drag to place a region of interest). Other userinteraction may be received, such as for activating the classification.

The processor 82 is a general processor, central processing unit,control processor, graphics processor, digital signal processor,three-dimensional rendering processor, image processor, applicationspecific integrated circuit, field programmable gate array, digitalcircuit, analog circuit, combinations thereof, or other now known orlater developed device for segmentation, extracting feature values,and/or classifying tissue. The processor 82 is a single device ormultiple devices operating in serial, parallel, or separately. Theprocessor 82 may be a main processor of a computer, such as a laptop ordesktop computer, or may be a processor for handling some tasks in alarger system, such as in an imaging system 80. The processor 82 isconfigured by instructions, design, hardware, and/or software to performthe acts discussed herein.

The processor 82 is configured to perform the acts discussed above. Inone embodiment, the processor 82 is configured to identify a region ofinterest based on user input, extract values for features for theregion, classify the tumor in the region (e.g., apply the machine-learntclassifier), and output results of the classification. In otherembodiments, the processor 82 is configured to transmit the acquiredframes of data or extracted values of features to the server 88 andreceive classification results from the server 88. The server 88 ratherthan the processor 82 performs the machine-learnt classification. Theprocessor 82 may be configured to generate a user interface forreceiving seed points or designation of a region of interest on one ormore images.

The display 86 is a monitor, LCD, projector, plasma display, CRT,printer, or other now known or later developed devise for outputtingvisual information. The display 86 receives images, graphics, text,quantities, or other information from the processor 82, memory 84,imaging system 80, or server 88. One or more medical images aredisplayed. The images are of a region of the patient. In one embodiment,the images are of a tumor, such as three-dimensional rendering of theliver with the tumor highlighted by opacity or color. The image includesan indication, such as a text, a graphic or colorization, of theclassification of the tumor. Alternatively or additionally, the imageincludes a quantity based on the classification, such as a tag value.The quantity or classification output may be displayed as the imagewithout the medical image representation of the patient. Alternativelyor additionally, a report with the classification is output.

The server 88 is a processor or group of processors. More than oneserver 88 may be provided. The server 88 is configured by hardwareand/or software to receive frames of data (e.g., multi-parametricimages), extracted features from frames of data, and/or other clinicalinformation for a patient, and return the classification. The server 88may extract values for the features from received frames of data. Toclassify, the server 88 applies a machine-learnt classifier to thereceived information. Where the classification identifies one or morereference cases similar to the case for a given patient, the server 88interacts with the database 90.

The database 90 is a memory, such as a bank of memories, for storingreference cases including treatments for tumor, frames of data and/orextracted values for features, and outcomes for evidence-based medicine.The server 88 uses the database 90 to identify the cases in the databasemost or sufficiently similar to a current case for a current patient.The server 88 transmits the identity of the reference and/or thereference information to the processor 82. In alternative embodiments,the server 88 and database 90 are not provided, such as where theprocessor 82 and memory 84 extract, classify, and output theclassification.

While the invention has been described above by reference to variousembodiments, it should be understood that many changes and modificationscan be made without departing from the scope of the invention. It istherefore intended that the foregoing detailed description be regardedas illustrative rather than limiting, and that it be understood that itis the following claims, including all equivalents, that are intended todefine the spirit and scope of this invention.

I (we) claim:
 1. A method for tissue characterization in medical imaging, the method comprising: scanning a patient with a medical scanner, the scanning providing multiple frames of data representing a tissue region of interest in the patient; extracting, by a processor, values for features from the frames of data; classifying, by a machine-learnt classifier implemented by the processor, a therapy response of the tissue of the tissue region from the values of the features as input to the machine-learnt classifier; and transmitting the therapy response.
 2. The method of claim 1 wherein scanning comprises scanning with a magnetic resonance scanner, computed tomography scanner, ultrasound scanner, positron emission tomography scanner, single photon emission computed tomography scanner, or combinations thereof.
 3. The method of claim 1 wherein scanning comprises scanning with a magnetic resonance scanner, the frames of data comprising functional measurements by the magnetic resonance scanner.
 4. The method of claim 1 wherein extracting values comprises extracting the values as scale-invariant feature transformation, histogram of oriented gradients, local binary pattern, gray-level co-occurrence matrix, or combinations thereof.
 5. The method of claim 1 wherein classifying comprises classifying from the values of the features extracted from the frames of data and values of clinical measurements for the patient.
 6. The method of claim 1 wherein scanning comprises scanning at different times, different contrasts, or different types, the frames corresponding to the different times, contrasts, or types, and wherein classifying comprises classifying the therapy response based due to differences in the frames.
 7. The method of claim 1 wherein extracting comprises extracting the values for the features with the features comprising deep-learnt features from deep learning, and wherein classifying comprises classifying by the machine-learnt classifier learnt with the deep learning.
 8. The method of claim 7 wherein extracting and classifying comprise extracting and classifying based on Siamese convolution networks.
 9. The method of claim 1 wherein classifying the therapy response comprises inferring therapy success for therapy applied to the tissue region.
 10. The method of claim 1 wherein classifying the therapy response comprises identifying an example treatment and outcome for another patient with similar extracted values of the features.
 11. The method of claim 1 wherein transmitting the therapy response comprises displaying the therapy response in a report.
 12. The method of claim 1 further comprising identifying, by a user input device responsive to user selection, the tissue region, wherein the extracting, classifying, and transmitting are performed without user input after the identifying.
 13. The method of claim 1 wherein the extracting and classifying are performed by the processor remote from the medical scanner.
 14. The method of claim 1, where therapy response comprises a responder classification, non-responder classification, or a prediction of survival time.
 15. The method of claim 1 wherein transmitting the therapy response comprises displaying the therapy response as an overlay over images of the tissue region.
 16. A method for tissue characterization in medical imaging, the method comprising: scanning a patient with a medical scanner, the scanning providing different frames of data representing different types of measurements for tissue in the patient; extracting, by a processor, values for features from the frames of data; classifying, by a machine-learnt classifier implemented by the processor, the tissue of the patient from the values of the features as input to the machine-learnt classifier; and transmitting the tissue classification.
 17. The method of claim 16 wherein scanning comprises scanning with a magnetic resonance scanner, the different types of measurements including an apparent diffusion coefficient.
 18. The method of claim 16 wherein extracting comprises extracting the values for a texture of the tissue represented by at least one of the frames of data.
 19. The method of claim 16 wherein classifying the tissue comprises classifying a predication of response of the tissue to therapy, classifying a benign or malignant, classifying a staging tag, a similarity to a case, or combinations thereof.
 20. The method of claim 16 wherein extracting comprises extracting the values for features that are deep-learnt features, and wherein classifying comprises classifying by the machine-learnt classifier being a deep-learnt classifier, the deep-learnt features and deep-learnt classifier being performed on multiple convolution networks.
 21. A method for tissue characterization in medical imaging, the method comprising: scanning a patient with a medical scanner, the scanning providing a frame of data representing a tumor in the patient; extracting, by a processor, values for deep-learnt features from the frame of data; classifying, by a deep-machine-learnt classifier implemented by the processor, the tumor from the values of the features as input to a machine-learnt classifier; and transmitting the classification of the tumor.
 22. The method of claim 21 wherein the deep-learnt features and the deep-machine-learnt classifier are from a Siamese convolution network. 